<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Using the FacemarkAAM</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d3/d81/tutorial_contrib_root.html">Tutorials for contrib modules</a></li><li class="navelem"><a class="el" href="../../d5/d47/tutorial_table_of_content_facemark.html">Tutorial on Facial Landmark Detector API</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Using the FacemarkAAM </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><h2>Goals </h2>
<p>In this tutorial you will learn how to:</p><ul>
<li>creating the instance of FacemarkAAM</li>
<li>training the AAM model</li>
<li>Fitting using FacemarkAAM</li>
</ul>
<h2>Preparation </h2>
<p>Before you continue with this tutorial, you should download the dataset of facial landmarks detection. We suggest you to download the LFPW dataset which can be retrieved at <a href="https://ibug.doc.ic.ac.uk/download/annotations/lfpw.zip">https://ibug.doc.ic.ac.uk/download/annotations/lfpw.zip</a>.</p>
<p>Make sure that the annotation format is supported by the API, the contents in annotation file should look like the following snippet: </p><div class="fragment"><div class="line">version: 1</div><div class="line">n_points:  68</div><div class="line">{</div><div class="line">212.716603 499.771793</div><div class="line">230.232816 566.290071</div><div class="line">...</div><div class="line">}</div></div><!-- fragment --><p>The next thing to do is to make 2 text files containing the list of image files and annotation files respectively. Make sure that the order or image and annotation in both files are matched. Furthermore, it is advised to use absolute path instead of relative path. Example to make the file list in Linux machine </p><div class="fragment"><div class="line">ls $PWD/trainset/*.jpg &gt; images_train.txt</div><div class="line">ls $PWD/trainset/*.pts &gt; annotation_train.txt</div></div><!-- fragment --><p>example of content in the images_train.txt </p><div class="fragment"><div class="line">/home/user/lfpw/trainset/100032540_1.jpg</div><div class="line">/home/user/lfpw/trainset/100040721_1.jpg</div><div class="line">/home/user/lfpw/trainset/100040721_2.jpg</div><div class="line">/home/user/lfpw/trainset/1002681492_1.jpg</div></div><!-- fragment --><p>example of content in the annotation_train.txt </p><div class="fragment"><div class="line">/home/user/lfpw/trainset/100032540_1.pts</div><div class="line">/home/user/lfpw/trainset/100040721_1.pts</div><div class="line">/home/user/lfpw/trainset/100040721_2.pts</div><div class="line">/home/user/lfpw/trainset/1002681492_1.pts</div></div><!-- fragment --><p>Optionally, you can create the similar files for the testset.</p>
<p>In this tutorial, the pre-trained model will not be provided due to its large file size (~500MB). By following this tutorial, you will be able to train obtain your own trained model within few minutes.</p>
<h2>Working with the AAM Algorithm </h2>
<p>The full working code is available in the face/samples/facemark_demo_aam.cpp file. In this tutorial, the explanation of some important parts are covered.</p>
<ol type="1">
<li><p class="startli"><b>Creating the instance of AAM algorithm</b></p>
<div class="fragment"><div class="line">    <span class="comment">/*create the facemark instance*/</span></div><div class="line">    <a class="code" href="../../d4/d41/namespacecv_1_1dynafu.html#afe352b7045bf5f1496c2d8dd2a09a764">FacemarkAAM::Params</a> <a class="code" href="../../d1/dae/namespacecv_1_1gapi_1_1ie.html#a3ab1729bcaf2d08e30dd2bc645410908">params</a>;</div><div class="line">    params.scales.push_back(2.0);</div><div class="line">    params.scales.push_back(4.0);</div><div class="line">    params.model_filename = <span class="stringliteral">&quot;AAM.yaml&quot;</span>;</div><div class="line">    Ptr&lt;FacemarkAAM&gt; facemark = FacemarkAAM::create(params);</div></div><!-- fragment --><p> Firstly, an instance of parameter for the AAM algorithm is created. In this case, we will modify the default list of the scaling factor. By default, the scaling factor used is 1.0 (no scaling). Here we add two more scaling factor which will make the instance trains two more model at scale 2 and 4 (2 time smaller and 4 time smaller, faster faster fitting time). However, you should make sure that this scaling factor is not too big since it will make the image scaled into a very small one. Thus it will lost all of its important information for the landmark detection purpose.</p>
<p class="startli">Alternatively, you can override the default scaling in similar way to this example: </p><div class="fragment"><div class="line">std::vector&lt;float&gt;scales;</div><div class="line">scales.push_back(1.5);</div><div class="line">scales.push_back(2.4);</div><div class="line"></div><div class="line">FacemarkAAM::Params params;</div><div class="line">params.scales = scales;</div></div><!-- fragment --></li>
<li><p class="startli"><b>Loading the dataset</b></p>
<div class="fragment"><div class="line">    <span class="comment">/*Loads the dataset*/</span></div><div class="line">    std::vector&lt;String&gt; images_train;</div><div class="line">    std::vector&lt;String&gt; landmarks_train;</div><div class="line">    <a class="code" href="../../db/d7c/group__face.html#ga02020fc9f387bb043a478fe5f112bb8d">loadDatasetList</a>(images_path,annotations_path,images_train,landmarks_train);</div></div><!-- fragment --><p> List of the dataset are loaded into the program. We will put the samples from dataset one by one in the next step.</p>
</li>
<li><p class="startli"><b>Adding the samples to the trainer</b></p>
<div class="fragment"><div class="line">    Mat image;</div><div class="line">    std::vector&lt;Point2f&gt; facial_points;</div><div class="line">    <span class="keywordflow">for</span>(<span class="keywordtype">size_t</span> i=0;i&lt;images_train.size();i++){</div><div class="line">        image = <a class="code" href="../../d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56">imread</a>(images_train[i].c_str());</div><div class="line">        <a class="code" href="../../db/d7c/group__face.html#gab70c6fb08756f867d6160099907202a5">loadFacePoints</a>(landmarks_train[i],facial_points);</div><div class="line">        facemark-&gt;addTrainingSample(image, facial_points);</div><div class="line">    }</div></div><!-- fragment --><p> The image from the dataset list are loaded one by one as well as its corresponding annotation data. Then the pair of sample is added to the trainer.</p>
</li>
<li><p class="startli"><b>Training process</b></p>
<div class="fragment"><div class="line">    <span class="comment">/* trained model will be saved to AAM.yml */</span></div><div class="line">    facemark-&gt;training();</div></div><!-- fragment --><p> The training process is called using a single line of code. Make sure that all the required training samples are already added to the trainer.</p>
</li>
<li><p class="startli"><b>Preparation for fitting</b></p>
<p class="startli">First of all, you need to load the list of test files. </p><div class="fragment"><div class="line">    <span class="comment">/*test using some images*/</span></div><div class="line">    <a class="code" href="../../dc/d84/group__core__basic.html#ga1f6634802eeadfd7245bc75cf3e216c2">String</a> testFiles(images_path), testPts(annotations_path);</div><div class="line">    <span class="keywordflow">if</span>(!test_images_path.empty()){</div><div class="line">        testFiles = test_images_path;</div><div class="line">        testPts = test_images_path; <span class="comment">//unused</span></div><div class="line">    }</div><div class="line">    std::vector&lt;String&gt; images;</div><div class="line">    std::vector&lt;String&gt; facePoints;</div><div class="line">    <a class="code" href="../../db/d7c/group__face.html#ga02020fc9f387bb043a478fe5f112bb8d">loadDatasetList</a>(testFiles, testPts, images, facePoints);</div></div><!-- fragment --><p> Since the AAM needs initialization parameters (rotation, translation, and scaling), you need to declare the required variable to store these information which will be obtained using a custom function. Since the implementation of getInitialFitting() function in this example is not optimal, you can create your own function.</p>
<p class="startli">The initialization is obtained by comparing the base shape of the trained model with the current face image. In this case, the rotation is obtained by comparing the angle of line formed by two eyes in the input face image with the same line in the base shape. Meanwhile, the scaling is obtained by comparing the length of line between eyes in the input image compared to the base shape.</p>
</li>
<li><p class="startli"><b>Fitting process</b></p>
<p class="startli">The fitting process is started by detecting the face in a given image. </p><div class="fragment"><div class="line">        image = <a class="code" href="../../d4/da8/group__imgcodecs.html#ga288b8b3da0892bd651fce07b3bbd3a56">imread</a>(images[i]);</div><div class="line">        myDetector(image, faces, &amp;face_cascade);</div></div><!-- fragment --><p> If at least one face is found, then the next step is computing the initialization parameters. In this case, since the getInitialFitting() function is not optimal, it may not find pair of eyes from a given face. Therefore, we will filter out the face without initialization parameters and in this case, each element in the <code>conf</code> vector represent the initialization parameter for each filtered face. </p><div class="fragment"><div class="line">            std::vector&lt;FacemarkAAM::Config&gt; conf;</div><div class="line">            std::vector&lt;Rect&gt; faces_eyes;</div><div class="line">            <span class="keywordflow">for</span>(<span class="keywordtype">unsigned</span> j=0;j&lt;faces.size();j++){</div><div class="line">                <span class="keywordflow">if</span>(getInitialFitting(image,faces[j],s0,eyes_cascade, R,T,scale)){</div><div class="line">                    conf.push_back(FacemarkAAM::Config(R,T,scale,(<span class="keywordtype">int</span>)params.scales.size()-1));</div><div class="line">                    faces_eyes.push_back(faces[j]);</div><div class="line">                }</div><div class="line">            }</div></div><!-- fragment --><p> For the fitting parameter stored in the <code>conf</code> vector, the last parameter represent the ID of scaling factor that will be used in the fitting process. In this example the fitting will use the biggest scaling factor (4) which is expected to have the fastest computation time compared to the other scales. If the ID if bigger than the available trained scale in the model, the the model with the biggest scale ID is used.</p>
<p class="startli">The fitting process is quite simple, you just need to put the corresponding image, vector of <code><a class="el" href="../../dc/d84/group__core__basic.html#ga11d95de507098e90bad732b9345402e8">cv::Rect</a></code> representing the ROIs of all faces in the given image, container of the landmark points represented by <code>landmarks</code> variable, and the configuration variables. </p><div class="fragment"><div class="line">            <span class="keywordflow">if</span>(conf.size()&gt;0){</div><div class="line">                printf(<span class="stringliteral">&quot; - face with eyes found %i &quot;</span>, (<span class="keywordtype">int</span>)conf.size());</div><div class="line">                std::vector&lt;std::vector&lt;Point2f&gt; &gt; landmarks;</div><div class="line">                <span class="keywordtype">double</span> newtime = (double)<a class="code" href="../../db/de0/group__core__utils.html#gae73f58000611a1af25dd36d496bf4487">getTickCount</a>();</div><div class="line">                facemark-&gt;fitConfig(image, faces_eyes, landmarks, conf);</div><div class="line">                <span class="keywordtype">double</span> fittime = ((<a class="code" href="../../db/de0/group__core__utils.html#gae73f58000611a1af25dd36d496bf4487">getTickCount</a>() - newtime)/<a class="code" href="../../db/de0/group__core__utils.html#ga705441a9ef01f47acdc55d87fbe5090c">getTickFrequency</a>());</div><div class="line">                <span class="keywordflow">for</span>(<span class="keywordtype">unsigned</span> j=0;j&lt;landmarks.size();j++){</div><div class="line">                    <a class="code" href="../../db/d7c/group__face.html#ga318d9669d5ed4dfc6ab9fae2715310f5">drawFacemarks</a>(image, landmarks[j],<a class="code" href="../../dc/d84/group__core__basic.html#ga599fe92e910c027be274233eccad7beb">Scalar</a>(0,255,0));</div><div class="line">                }</div><div class="line">                printf(<span class="stringliteral">&quot;%f ms\n&quot;</span>,fittime*1000);</div><div class="line">                <a class="code" href="../../d7/dfc/group__highgui.html#ga453d42fe4cb60e5723281a89973ee563">imshow</a>(<span class="stringliteral">&quot;fitting&quot;</span>, image);</div><div class="line">                <a class="code" href="../../d7/dfc/group__highgui.html#ga5628525ad33f52eab17feebcfba38bd7">waitKey</a>(0);</div><div class="line">            }<span class="keywordflow">else</span>{</div><div class="line">                printf(<span class="stringliteral">&quot;initialization cannot be computed - skipping\n&quot;</span>);</div><div class="line">            }</div></div><!-- fragment --><p> After the fitting process is finished, you can visualize the result using the <code>drawFacemarks</code> function. </p>
</li>
</ol>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:38 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
